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ABSTRACT 



Based on an extensive analysis of the characteristics of 113 
accountability reports from 32 state education agencies, a study group 
convened to develop a list of necessary, desirable, and succinct 
characteristics of good state and district educational accountability 
reports. This report discusses the recommendations of the study group. 
Desirable characteristics were identified for both content and format, such 
as: (1) be clear about who the report is directed to, the intended purposes 

of the report, and the state's conceptual model for its accountability 
system; (2) be comprehensive yet concise in the reporting of inputs, 
processes, and results for students, especially students with disabilities; 

(3) provide comparative information with changes over time between schools, 
districts, states, regions, or standards,* (4) ensure reports are concise or 
contain carefully chosen indicators so that no more information is given than 
is necessary; (5) include cautions against misinterpretations of the data or 
against any unintended consequences; (6) make efforts to maintain 
confidentiality and avoid the possibility of identifying individual students; 
(7) use good formats so that reports are well-organized, readable, and 
interesting. A checklist of desirable characteristics for state and school 
district accountability reports is included for those who develop reports. 
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Executive Summary 






State accountability reports are designed to inform the public about the 
status of schools and their students. Across the United States, there is 
tremendous variabihty in what these reports look like, in the information 
they include, and in the extent to which they include students with 
disabihties. Because of this variabihty and the desire to produce quahty 
reports that would include data on students with disabihties, a work 
group was formed to develop a hst of necessary, desirable, and succinct 
characteristics of good state and district educational accountability 
reports. 

Desirable characteristics were identified for both content and format. 
They included: 

• Be clear about who the report is directed to, the intended purposes 
of the report, and the state’s conceptual model for its accountabihty 
system. 

• Be comprehensive yet concise in the reporting of inputs, processes, 
and results for students, especiaUy students with disabihties. 

• Provide comparative information with changes over time between 
schools, districts, states, regions, or standards. 

• It is strongly recommended that reports be concise or contain 
carefully chosen indicators so that no more information is given 
than is necessary. A multi-layered approach may be appropriate. 

• Include cautions against misinterpretations of the data or against 
any unintended consequences. 

• All efforts should be taken to maintain confidentiality and avoid 
the possibihty of identifying individual students. 

• Use good formats so that reports are weU-organized, readable, and 
interesting. Use catchy titles, pictures, or other visual aids to capture 
and hold the audience’s attention. 

A checkhst of desirable characteristics for state and school district 
accountabihty reports is included for those who develop reports. 
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Overview 



Increasingly, state education departments are recognizing that inclusive 
accountability systems and the pubhc reporting of educational results 
for students are important tools to help students attain higher educational 
standards. Many states have already designed accountabihty systems 
to ensure those who are inside and outside the educational system that 
students are moving toward desired goals (Brauen, O’Reilly, & Moore, 
1992). With recent federal mandates, such as the passage of RL. 105- 
17, the reauthorization of the Individuals with Disabihties Education 
Act (IDEA), state education agency personnel are required to report 
annually on the performance and progress of students with disabihties. 

States, however, vary greatly in their reporting practices (Elliott, 
Thurlow, & Ysseldyke, 1996; Thurlow, Langenfeld, Nelson, Shin, & 
Coleman, 1997). Some states exclude students with disabihties from 
their accountabihty reports, others exclude students who take tests with 
accommodations, while others exclude those who have used specific 
types of accommodations. 

Similarly, state accountabihty reports vary. Some states produce five 
or six 500-page volumes annually, while others produce a two- to three- 
page report. Consumers indicate that there is often too much information, 
or too httle. A few states give state level data; others give school, district, 
and state level data annually. Accountabihty documents also vary in 
their focus. Almost all states report on performance indicators for 
students in regular education, but very few include performance data 
for students with disabihties. Many states use these reports for 
accreditation purposes while others use them for technical assistance, 
diplomas, comphance with state requirements, or to generate local, 
district, and national comparisons. The indicators used in accountabihty 
reporting also cover a wide spectrum, from detailed financial 
information to student mobihty rates, and from staffing information to 
minutes spent in math and reading. Some states report numbers of 
students who met state standards or goals; others do not. Many states 
use tables or spreadsheets, and some even use the Internet to 



Some states 
exclude students 
with disabilities 
from their 
accountability 
reports, others 
exclude students 
who take tests 
with 

accommodations, 
while others 
exclude those who 
have used specific 
types of 

accommodations. 
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communicate their educational results. As one can see, the variabihty 
in reporting is considerable; it would be ideal if there was consistency 
in reporting practices. 



State assessment 
and special 
education 
representatives 
convened to 
develop a list of 
necessary, 
desirable, and 
succinct 
characteristics of 
good state and 
district 
accountability 
reports. 



With this variabihty in reporting in mind, members of a study group 
within the State Collaborative on Assessment and Student Standards, 
Assessing Special Education Students, which is made up of assessment 
and special education representatives, convened to develop a hst of 
necessary, desirable, and succinct characteristics of good state and 
district educational accountabihty reports. Those in attendance included 
representatives from six states (Alaska, Arkansas, Cahfomia, Colorado, 
Missouri, Wyoming), one Regional Resource Center (Mid-South), 
University personnel (Minnesota, Oregon), assessment personnel from 
the Council of Chief State School Officers (CCSSO), and personnel 
from the National Center on Educational Outcomes (NCEO). 

This meeting was devoted primarily to describing current accountabihty 
reports, discussing issues encountered in reporting, and generating 
characteristics of good reports. Before the meeting, NCEO personnel 
had conducted an extensive analysis of the characteristics of 1 13 reports 
received from 32 state education agencies (SEAs). Among highhghts 
of that analysis were the findings on length and variabihty described in 
the introduction to this paper, and many others, including those hsted 
in Table 1. 

The study group first generated a hst of characteristics of good reports, 
then carefully described the way characteristics could apply to reports 
issued to differing audiences. 
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Table 1. Highlights of Findings in Analysis of State and District 

Educational Accountability Reports 

• Reports vary in length, from two pages to approximately 
600 pages. 

• Reports vary in format (spreadsheets, data tables, bar 
graphs, narratives, Internet). 

• It is rare to find information on the intended purpose of a 
report or its intended audience. 

• The most common target audiences of reports, when 
identified, are the general pubUc and government agencies. 

• The most common purpose of reports, when identified, is 
“to provide information.” 

• The majority of states provide multiple accountabiUty 
documents, but 16 states compile their data into a single 
document. 

• Most states provide data at both the state and district levels 
(N=41). 

• All states report outcome data, and most report input and 
process data. 

• Few states report outcome data for students with 
disabiUties. 

• Every state has at least one report in which there are data 
reported for a sample that is not described clearly. 

• Eight states have at least one report in which the report 
specifically excluded the outcome data of students with 
disabilities. 
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Results 
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Participants 
indicated that it 
should be clear 
who the report is 
directed to, the 
intended purpose 
of the report, and 
the state’s 
conceptual model 
for its 
accountability 
system. 



The SCASS Study Group generated various characteristics of good 
accountabihty reports. These characteristics were grouped, formatted, 
and developed into an easy-to-use checklist for state education 
department personnel. The generated characteristics were placed into 
three broad categories: content, format, and general. Below are the 
characteristics the study group found to be most desirable in creating 
effective accountabihty reports. 

Be clear. Participants indicated that it should be clear who the report is 
directed to, the intended purpose of the report, and the state’s conceptual 
model for its accountabihty system. Other desired aspects of clarity 
were a clear statement of mission or goals, assumphons, and standards. 
Finally, it was thought that the nature of the populahon of students 
being reported on should be clearly described. 

Be comprehensive. State accountability reports should be 
comprehensive, yet concise in their reporting of inputs, processes, and 
results. “Inputs” refers to accountabihty indicators that describe the 
student’s learning environment as well as demographic characterishcs 
of the particular district’s populahon (e.g., student-teacher raho, cost). 
“Process” indicators describe student participahon and school district 
evaluahon (e.g., enrollment, attendance, accreditahon status). “Results” 
refer to nontest and test data indicators that focus on the end result of a 
student’s learning process. Data should include all three types of 
indicators, and should be reported for all students, especially students 
with disabihhes. It would be most informahve if the results for special 
populahons were disaggregated. With the reauthorizahon of IDEA, 
states are required to disaggregate the performance data of students 
with disabihhes. 




Provide comparative information. Comparahve informahon should 
be given, but with cauhons. For instance, if data were not collected on 
similar populahons, this should be made known. It should be possible 
to make fair comparisons from the data reported. Not only should 
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comparisons be fair, but it is important to make regional and national 
comparisons to check student progress. If comparisons should not be 
drawn, then it is the responsibility of the state to make this clear to its 
audience. 



Be concise. Group participants strongly indicated that reports should 
be concise. Some states have been known to produce three or four 
volumes with over 500 pages of data in each volume. Indicators need 
to be chosen carefully so that no more information is given than is 
necessary to convey a message to an intended audience. It was suggested 
that a multi-layered approach may be appropriate for those states that 
have a large amount of information to report. For a consumer who might 
not need all the available details, an abbreviated report would be 
sufficient. 

Include cautions. SEA or local education agency (LEA) personnel 
should always include cautions against misinterpretation of data or 
against any unintended consequences of the data. Reports should be 
written in a way that minimizes scapegoating and negatives even when 
presenting bad news. 

Maintain confidentiality. Participants indicated concern about 
maintaining confidentiality for low-incidence student populations. All 
efforts should be taken to avoid the possibility of identifying individual 
students. This may involve reporting data of certain special populations 
only at the state or district levels. 

Use good formats. Participants generated four kinds of desirable format 
characteristics: 



State and district 
reports should 
include cautions 
against 

misinterpretation 
of data or against 
any unintended 
consequences of 
the data. 



• Accountability reports should be readable by the intended 
audience, and should be responsive to the needs of the 
intended audience. 

• The layout should be eye-catching or interesting, organized 
and contain one concept per page. Organizational aids should 
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be used, and might include a reader’s guide, table of contents, 
and/or index or glossary. 

• A bulletted summary of the report for a quick read is helpful 
to consumers. 

• The report should make it clear how additional copies may 
be obtained or how a person could obtain more detailed 
information if desired. 



Summary 

Now that accountability systems and reporting of educational results 
have moved to the front burner for state education departments, it is 
important to examine how accountability reports are presented. Neither 
500-page reports nor 2-page surmnaries properly meet all education 
and public needs. The above recommendations can help develop 
templates for reports and enhance cormnunication to stakeholders about 
the educational results for students, with and without disabilities. 
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Checklist of Guidelines for State and District 
Educational Accountability Reports 
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Checklist of Guidelines for State and District Educational 
Accountability Reports 



CONTENT 

Clear... 

□ Clear statement of intended audience 

□ Clear statement of intended purpose 

□ Clear statement of states’ conceptual model for its accountability 
system (including inputs, processes, and results) 

□ Clear statement of state standards (or goals) or mission/vision 

□ Clear statement of assumptions 

□ Clear statement of who was included in the population of students 
being reported on 

Comprehensive... 

□ Comprehensive, yet concise set of inputs, processes, and results 

□ Data on aU students, including students with disabilities and limited 
English proficient students (students with disabilities and limited 
English proficient students’ results are disaggregated) 

Comparative... 

Includes enough information to enable people to make fair 
comparisons among: 

□ Schools 

□ Districts 

□ States 

□ Regions 

□ Standards 

Includes enough information to enable people to make fair 
judgments about changes over time for: 

□ Schools 

□ Districts 

□ States 



Concise... 

□ Includes no more information than is necessary to convey a message 
to an intended audience ... brief 
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Cautions... 

Provides cautions against: 

□ Scapegoating 

□ Unintended consequences 

□ Negatives 

Confidentiality... 

□ Maintains confidentiality of low frequency student populations 

FORMAT 

Readable... 

□ Appropriate for the intended audience 

Responsive to the needs of intended audiences... 

□ Answers audience questions and provides accurate profile 

Layout... 

□ Not cluttered or complex 

□ Organized and easy to find information (e.g., reader’s guide, table 
of contents, index, glossary) 

□ Interesting (e.g., includes catchy titles, pictures, or other devices to 
get and hold audience interest) 

Links... 

□ Statement of how and where to get additional copies 

□ Statement of how to get more detailed information 

Executive Summary... 

□ BuUetted summary of report for a “quick read” 



IMPORTANT OVERALL QUESTIONS 



Is the report readable? 


□ YES 


□ NO 


Is the report fair? 


□ YES 


□ NO 


Is the report concise? 


□ YES 


□ NO 


Is the report visually attractive? 


□ YES 


□ NO 


Is the report accurate? 


□ YES 


□ NO 



12 



17 



NCEO 






The College of Education 
llllil &L Human Development 

University of Minnesota 

18 




. m 



EC 30(^151 



* ^ 




as, DEPARTMENT OF EDUCATION 

OtHce of Educational Research and Improvement (OERI) 
Educational Resources Information Center (ERIC) 




NOTICE 

RF.PRonnrTroN basis 




Tliis document is covered by a signed “Reproduction Release 
(Blanket)” fomi (on fde within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 




This document is Federally-funded, or carries its own perrrussion to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release 
form (either “Specific Document” or “Blanket”). 



ERIC 



i 







